Image editing on a wearable device

ABSTRACT

Image editing on a wearable device includes a system which obtains sensor data via the wearable device. The sensor data includes a representation of hand movement, head movement or voice command associated with a user. The system executes an application for editing an image based on the obtained sensor data. The system provides for display a list of image adjustment types associated with the application. The system selects an image adjustment type based on one or more of the hand movement, the head movement or the voice command. The system provides for display a prompt having options to adjust a property of the selected image adjustment type. The system selects one of the options included in the prompt. The system modifies an image based on the selected option. The system then provides the modified image for storage in a data structure of a memory unit in the wearable device.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. Provisional PatentApplication No. 62/030,502, entitled “Method and System for EditingImages on a Head Wearable Device,” filed Jul. 29, 2014, the contents ofwhich are hereby incorporated by reference in its entirety for allpurposes.

BACKGROUND

A wearable computing device is intended to be worn on the body of auser, such as the user's head, and displays information in the form ofimages within a field of view of one user's eye, referred to asmonocular wearable device, or within the field of view of each user'seye, referred to as binocular wearable device. Wearable display devicescan provide a mobile platform to communicating, computing andinteracting with a surrounding environment such that the wearabledisplay devices may be applicable to a wide range of applications thataugment a user's experience of the surrounding environment.

SUMMARY

The present description relates generally to wearable computing devices,and more particularly, but not exclusively, to image editing on wearabledevices.

In an embodiment, a computer-implemented method includes obtaining imageinformation by a sensor unit of a wearable device, the image informationincluding a representation of a moveable object associated with a user.The method includes executing an application for editing an image by aprocessing unit of the wearable device, the application being executedbased on the image information. The method includes providing fordisplay, on a display unit of the wearable device, a list of imageadjustment types associated with the application. The method includesselecting at least one image adjustment type of the list of imageadjustment types based on a first position of the moveable object. Themethod includes providing for display, on the display unit, a promptincluding levels of adjustment for the selected at least one imageadjustment type. The method includes selecting a level of adjustmentbased on a second position of the moveable object. The method includesmodifying an image using the selected level of adjustment. The methodfurther includes providing the modified image for display on the displayunit.

In another embodiment, a non-transitory machine-readable mediumembodying instructions, which when executed by one or more processors,cause a computing device to perform operations. The operations includereceiving, at a processor unit of a wearable device, first audio datafrom a sensor unit of the wearable device, in which the first audio datais associated with a user. The operations include providing for display,on a display unit of the wearable device, a user interface of anapplication for editing an image based on a first user input in thefirst audio data. The operations include providing for display, on thedisplay unit via the user interface, a list of image adjustment types.The operations include selecting at least one image adjustment type ofthe list of adjustment types based on a second user input via the userinterface. The operations include providing for display, on the displayunit via the user interface, a prompt including levels of adjustment forthe selected at least one image adjustment type, in which each of thelevels of adjustment is displayed with a label corresponding to a levelof adjustment. The operations include selecting a level of adjustmentbased on a third user input via the user interface. The operationsinclude modifying an image based on the selected level of adjustment.The operations further include providing the modified image for displayon the display unit.

In still another embodiment, an apparatus includes a display unit, oneor more sensor units, one or more memory units and one or moreprocessing units configured to execute executable instructions toperform operations. The operations include receiving, at the one or moreprocessor units, first sensor data from the one or more sensor units,the first sensor data comprising first image information having arepresentation of a moveable object associated with a user. Theoperations include identifying one or more first movements by the objectincluded in the first image information, in which the one or more firstmovements include one or more of a head movement by the user, a handmovement by the user or a gaze direction by the user. The operationsinclude determining that the one or more first movements correspond to afirst gesture associated with a first user input. The operations includeproviding for display, on the display unit, a user interface of anapplication for editing an image based on the first user input. Theoperations include providing for display, on the display unit via theuser interface, a list of image adjustment types. The operations includedetermining a position of the first gesture relative to the list ofimage adjustment types. The operations include determining a matchbetween a position of at least one image adjustment type of the list ofadjustment types and the determined position of the first gesture. Theoperations include selecting the at least one image adjustment typebased on the determined match. The operations include providing fordisplay, on the display unit via the user interface, a prompt includinga plurality of levels of adjustment for the selected at least one imageadjustment type. The operations include receiving, at the one or moreprocessor units, second sensor data from the one or more sensor units inresponse to the prompt provided for display, in which the second sensordata includes second image information relating to the moveable object.The operations include determining one or more second movements by themoveable object included in the second image information, in which theone or more second movements include one or more of the head movement orthe hand movement by the user. The operations include determining thatthe one or more second movements correspond to a second gestureassociated with a second user input. The operations include selecting alevel of adjustment for the selected at least one image adjustment typebased on the second user input. The operations include modifying animage based on the selected level of adjustment. The operations furtherinclude providing the modified image for display on the display unit.

The description in this summary section may provide some illustrativeexamples of the disclosure. This section is not intended to be a broadoverview or to identify essential elements of the disclosure.

It is understood that other configurations of the subject technologywill become readily apparent to those skilled in the art from thefollowing detailed description, in which various configurations of thesubject technology are shown and described by way of illustration. Aswill be realized, the subject technology is capable of other anddifferent configurations and its several details are capable ofmodification in various other respects, all without departing from thescope of the subject technology. Accordingly, the drawings and detaileddescription are to be regarded as illustrative in nature and not asrestrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide furtherunderstanding and are incorporated in and constitute a part of thisspecification, illustrate disclosed implementations and together withthe description serve to explain the principles of the disclosedimplementations. In the drawings:

FIG. 1 illustrates an example wearable display device in which imageediting may be implemented.

FIG. 2 is a block diagram illustrating an example computer system withwhich some implementations of the subject technology can be implemented.

FIG. 3 illustrates an example process for image editing using handgestures detected by the wearable display device of FIG. 1 according tocertain aspects of the disclosure.

FIG. 4 illustrates an example process for image editing using headmovement detected by the wearable display device of FIG. 1 according tocertain aspects of the disclosure.

FIG. 5 illustrates an example process for image editing using audiblecommands detected by the wearable display device of FIG. 1 according tocertain aspects of the disclosure.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth to provide a full understanding of the present disclosure. It willbe apparent, however, to one ordinarily skilled in the art thatimplementations of the present disclosure may be practiced without someof these specific details. In other instances, well-known structures andtechniques have not been shown in detail so as not to obscure thedisclosure. Reference is made in the following detailed description tothe accompanying figures, which form a part thereof. In the figures,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description and figures are not meant to be limiting. Otherembodiments may be utilized, and other changes may be made, withoutdeparting from the spirit or scope of the subject matter presentedherein. It will be readily understood that the aspects of the presentdisclosure, as generally described herein, and illustrated in thefigures, can be arranged, substituted, combined, separated, and designedin a wide variety of different configurations, all of which arecontemplated herein.

FIG. 1 illustrates an example wearable display device 100 in which imageediting may be implemented. The wearable display device 100 integratesvarious elements, such as input devices, sensor devices, detectordevices, display units, wireless communication devices as well as imageand audio processors. Not all of the depicted components may berequired, however, and one or more implementations may includeadditional components not shown in the figure. Variations in thearrangement and type of the components may be made without departingfrom the scope of the claims as set forth herein. Additional components,different components, or fewer components may be provided.

The wearable display device 100 is intended to be worn by a userincluding, but not exclusively to, the head of the user, a waist of theuser, or any other member of the user's body. The wearable displaydevice 100 includes a first frame 110 and a second frame 120, displayunits 112 and 122, a frame support structure 118, and frame stems 130and 140. The wearable display device 100 also includes a first cameradevice 114 and a second camera device 116 on the first frame 110. Thewearable display device 100 also includes motion sensor devices 134 and136 and an image projection device 132 on the frame stem 130. Thewearable display device 100 also includes a microphone device 142, aninput device 144 and a computing device 146 on the frame stem 140. Insome aspects, the wearable display device 100 is a multi-componentdevice such that a first component is communicably coupled to a secondcomponent to facilitate transfer of data between the two components. Forexample, the first component may be worn on the user's head and thesecond component may be worn on the user's waist; however, the number ofcomponents and/or location of such components may vary depending onimplementation.

The first camera 114 is coupled to an outer surface of the first frame110 and is proximally located to the display unit 112. Similarly, thesecond camera 116 is coupled to an inner surface of the first frame 110and is proximally located to the display unit 112. The first cameradevice 114, referred to as a field of view (FOV) camera, may be a camerathat captures images and/or video within a FOV of the user wearing thewearable display device 100. The second camera device 116, referred toas facing an eye (FEY) camera, may be a camera that captures imagesand/or video of the user's eye. The first camera device 114 and thesecond camera device 116 may be charge-coupled device (CCD) cameras orcomplementary metal-oxide-semiconductor (CMOS) cameras, depending onimplementation. As known to one of ordinary skill in the art, some CCDand CMOS cameras typically use the same kind of vision element, namely aphoto diode.

The display units 112 and 112 are referred to as separate displays insome embodiments, or referred to as a common display in otherembodiments. The display units 112 and 122 are housed within the firstand second frames 110 and 120, respectively. The display units 112 and122, referred to as lenses, may provide for display images and/or videoto the user. The images and/or video may be displayed within the FOV ofthe user. Example display units 112, 122 include display devices, suchas a LED (light emitting diode) screens or LCD (liquid crystal display)screens, for displaying information to the user. The images and/or videofor display on the display units 112 and 122 originate from thecomputing device 146 in some embodiments, or originate from an externalsource communicably coupled to the wearable display device 100 in otherembodiments.

The image projection device 132 is coupled to an inner surface of theframe stem 130 and is proximally located to the display unit 112. Theimage projection device 132 may project a display onto an inner surfaceof the display unit 112. In some aspects, the display units 112, 122include a region with an opaque reflective surface for displaying imagesprojected by the image projection device 132.

The motion sensor devices 134 and 136 are housed within the frame stem130. The motion sensor device 134 may be an accelerometer for measuringlinear acceleration of the wearable display device 100, and the motionsensor device 136 may be a gyroscope for measuring linear and/orrotational movements by the wearable display device 100. In someaspects, the motion sensor devices 134 and 136 may be micro-sizedpiezoelectric sensors. In other aspects, the motion sensor devices 134and 136 may be micro-electrical-mechanical system (MEMS) sensors. Eachof the motion sensor devices 134 and 136 may be a single component, ormay include more than one component.

The wearable display device 100 may be secured onto a user's face by theframe support structure 118 coupled between the first and second frames110 and 120, which are coupled to the frame stems 130 and 140,respectively. The wearable display device 100 may be formed as a solidstructure composed of bulk material, including but not limited to, oneor more of plastic, metal, ceramic, or glass. The wearable displaydevice 100 may be formed as a hollow structure composed of the bulkmaterial or variation thereof to allow wiring and componentinterconnections to be internally routed through the wearable displaydevice 100. Each of the frame stems 130 and 140 is an example of aprojection that extends away from a display (e.g., display units 112 and122) and is used to secure the display to the user.

The computing device 146 may include one or more processor units (notshown) and one or more memory units (not shown). In one or moreimplementations, the computing device 146 is integrated into one of theframe stems 130 and 140 of the wearable display device 100. In someaspects, the computing device 146 may communicate with an externalcomputing device mounted to a headgear such as a hat or a helmet,referred to as a head-mounted computing device, or mounted to a memberof the user such as an arm, leg or waist of the user, referred to as abody-mounted computing device.

The computing device 146 may be operably coupled to the first cameradevice 114, the second camera device 116, the motion sensor devices 134and 136, the microphone device 142, and the image projection device 132.The computing device 146 may receive and process input sensor data fromone or more of the first camera device 114, the second camera device116, the motion sensor devices 134 and 136, and the microphone device142 in order to send image data to the image projection device 132 fordisplay on at least the display unit 112. The input device 144 may be atouch pad for the user to send one or more inputs to the computingdevice 146. The microphone device 142 is coupled to a surface of theframe stem 140 and is proximally located to the display unit 122. Themicrophone device 142 may be housed within the frame stem 140 such thata portion of the microphone device 142 is exposed on a surface of theframe stem 140. The input device 144 may be housed within the frame stem140 and may include one or more input ports on a surface of the framestem 140 for accepting user input via an external device. The computingdevice 146 is housed within the frame stem 140.

By way of example and without limitation, the wearable display device100 may include a magnetometer and another orientation sensor device(e.g., a theodolite). The magnetometer may measure the Earth's magneticfield, which combined with a gravity measurement by the motion sensordevice 136, can be used to determine an orientation of the wearabledisplay device 100 in two- or three-dimensions. The theodolite can be amulti-function viewfinder that combines a compass, a two-axisinclinometer, and a rangefinder into one application.

During operation, the computing device 146 may monitor hand gesture dataprovided by the first camera device 114, eye movement data provided bythe second camera device 116, head movement data provided by the motionsensor devices 134 and 136, and audio data provided by the microphonedevice 142 to modify images displayed on the inner surface of thedisplay unit 112 as intended by the user.

The editing of digital image content or video content on the wearabledisplay device 100 may be difficult because conventional editingelements such as a mouse or a touchscreen are unavailable on suchwearable computing devices. In this regard, there is a need for acontrol scheme or method that enables a user to 1) edit images directlyafter they have been captured with the wearable display device 100, 2)edit images taken by a camera external to the wearable display device100, or 3) edit images downloaded by the wearable display device 100.This may be the case for users who need to concentrate on more than onetask at a time in a professional environment and the like, such as apilot, a medical surgeon, or a bird photographer out in the wilderness.There also may be time-sensitive situations that rule out thepossibility of an upload of the captured images to a network, such as acloud network, and editing the uploaded images on a different computingdevice.

In accordance with the present disclosure, editing images on thewearable display device 100 can be performed by one or more handgestures, by one or more head gestures, by audible voice commands, andby one or more eye movements. Accordingly, the wearable display device100 is equipped with one or more cameras that can record hand gestures,with one or more gyroscopes/accelerometers that sense head gesturesand/or head movements, with one or more voice capture units that detectvoice commands, and with one or more eye gazing cameras that captureimages of eye movements by the person wearing the wearable displaydevice 100, thereby enhancing the user experience of editing images viathe wearable display device 100.

In one or more implementations, hand gestures are detected to provideuser input to the wearable display device 100 to edit images. Thewearable display device 100, which includes the display units 112 and122 and the first camera device 114, may recognize predetermined handgestures and carry out editing actions in response. A predetermined handgesture, such as a basic hand posture, may be used for activating animage editing user interface. For example, an index finger of one handmay be elongated (the remaining fingers being bent or closed) andpositioned within a field of view of the first camera device 114. Withthe hand in the basic hand posture, detecting the index finger moving upand down corresponds to a user input that enables a selection of a typeof image adjustment. The selection may be made from a displayedlist-like menu of image adjustments containing, for example, contrast,brightness, saturation adjustments to the image, or other parametervalues of filtration suitable for the image. In some aspects, the imageadjustments include changing the location of the light source. Forexample, the light source within the image may be adjusted from abottom-right corner to an upper-left corner in the image. In addition,detecting the user's hand movement and/or the index finger movement tothe left and/or right corresponds to an input to adjust the strength ofthe selected adjustment. In one embodiment, finishing the adjustment,e.g., disabling the image editing user interface, can be accomplished bychanging the hand posture, for example, by closing the hand or openingthe hand entirely.

In one or more implementations, head gestures are detected to provideuser input to the wearable display device 100 to edit images. Thewearable display device 100, which includes the display units 112 and122 and the second camera device 116, may recognize predetermined headgestures and carry out editing actions in response. A predetermined headgesture, such as a basic head tilt, may be used for activating the imageediting user interface. For example, a sequence of the user's headraising and lowering may correspond to a user input to activate theinterface. In some aspects, a movement of the user's head in a verticaldirection may select a corresponding type of adjustment from thelist-like menu described above. If one of the adjustments is selected,detection of the user's head moving in a leftward direction may decreasethe strength of the adjustment, whereas detecting the user's head movingin a rightward direction may increase the strength of the adjustment. Insome embodiments, the adjustment sequence may terminate (e.g., disablingthe image editing user interface) in response to detecting the user'shead tilting toward a detectable angle such as to the left or to theright.

In one or more implementations, voice commands/inputs are detected toprovide user input to the wearable display device 100 to edit images.The wearable display device 100, which includes the display units 112and 122 and a voice/audio detecting unit associated with the microphonedevice 142, may recognize predetermined voice commands and carry outediting actions in response. In some aspects, the predetermined voicecommands may be detected as a sequence of words, phrases and/orstatements. For example, a sequence of voice commands may include afirst statement such as “start image editing,” followed by a secondstatement such as “select adjustment” and then followed by one of“contrast”, “brightness”, and “saturation” statements as choices forselecting a type of image adjustment. Subsequently, for changing thestrength of the selected image adjustment, a corresponding voice commandmay include the phrase “adjust strength” followed by the word “more” or“less,” or in the alternative, a corresponding numerical value. In someaspects, the image adjustment may be terminated by detecting the word“stop” or a statement such as “stop editing.” The foregoing phrases andcorresponding words are merely examples, and other words, phrases orstatements may be used to represent the predetermined voice commands.

In one or more implementations, eye movements are detected to provideuser input to the wearable display device 100 to edit images. Thewearable display device 100, which includes the display units 112 and122 and the second camera device 116 may recognize predetermined eyemovements and carry out editing actions in response. A predetermined eyemovement, such as a forward-looking eye gaze for a predetermined periodof time, may be used for activating the image editing user interface.Alternatively, a sequence of shifting by the user's eye in aleft-to-right direction or a right-to-left direction may be interpretedto activate the image editing user interface. In some aspects, an imageadjustment is triggered by a detection of a gazing of the eye on aspecific location of the list-like menu containing the contrast,brightness, and saturation adjustments. Once the image adjustment isselected, detecting the user's eye moving in a leftward direction maydecrease the strength of the image adjustment, whereas detecting theuser's eye moving in a rightward direction may increase the strength ofthe image adjustment. The image adjustment may be terminated bydetecting the user's eye closing a corresponding eyelid.

FIG. 2 is a block diagram illustrating an example computer system 200with which some implementations of the subject technology can beimplemented. In certain aspects, the computer system 200 may beimplemented using hardware or a combination of software and hardware,either in a dedicated computing device, or integrated into anotherentity, or distributed across multiple entities.

The computer system 200 includes the display unit 112, the display unit122, the first camera device 114, referred to as the FOV camera, thesecond camera device 116, referred to as the FEY camera, the imageprojection device 132, the motion sensor device 134, referred to as theaccelerometer, the motion sensor device 136, referred to as thegyroscope, the microphone device 142, the input device 144 and thecomputing device 146.

The computing device 146 includes a processor unit 202, a communicationunit 204 and a memory unit 206. The first camera device 114, the secondcamera device 116, the image projection device 132, the motion sensordevice 134, the motion sensor device 136, the microphone device 142 andthe input device 144 are coupled to the computing device 146, which arethus operatively connected to the processor unit 202, which in turn areoperably coupled to or integral with the memory unit 206.

The computer system 200 may include a bus (not shown) or othercommunication mechanism for communicating information, and a processor(e.g., the processor unit 202) coupled with the bus for processinginformation. By way of example, the computer system 200 may beimplemented with one or more processor units 202. The processor unit 202may be a general-purpose microprocessor, a microcontroller, a DigitalSignal Processor (DSP), an Application Specific Integrated Circuit(ASIC), a Field Programmable Gate Array (FPGA), a Programmable LogicDevice (PLD), a controller, a state machine, gated logic, discretehardware components, or any other suitable entity that can performcalculations or other manipulations of information.

The computer system 200 can include, in addition to hardware, code thatcreates an execution environment for the computer program in question,e.g., code that constitutes processor firmware, a protocol stack, adatabase management system, an operating system, or a combination of oneor more of them stored in an included memory (e.g., the memory unit206), such as a Random Access Memory (RAM), a flash memory, a Read OnlyMemory (ROM), a Programmable Read-Only Memory (PROM), an Erasable PROM(EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, orany other suitable storage device, coupled to the bus for storinginformation and instructions to be executed by the processor unit 202.The processor unit 202 and the memory unit 206 can be supplemented by,or incorporated in, special purpose logic circuitry.

In one or more implementations, the memory unit 206 stores informationand instructions executable by the processor unit 202 or other processorunits and/or applications associated with the memory unit 206. Theinstructions may be stored in the memory unit 206 and implemented in oneor more computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer readable medium for executionby, or to control the operation of, the computer system 200, andaccording to any method well known to those of skill in the art,including, but not limited to, computer languages such as data-orientedlanguages (e.g., SQL, dBase), system languages (e.g., C, Objective-C,C++, Assembly), architectural languages (e.g., Java, .NET), andapplication languages (e.g., PHP, Ruby, Perl, Python). Instructions mayalso be implemented in computer languages such as array languages,aspect-oriented languages, assembly languages, authoring languages,command line interface languages, compiled languages, concurrentlanguages, curly-bracket languages, dataflow languages, data-structuredlanguages, declarative languages, esoteric languages, extensionlanguages, fourth-generation languages, functional languages,interactive mode languages, interpreted languages, iterative languages,list-based languages, little languages, logic-based languages, machinelanguages, macro languages, metaprogramming languages, multiparadigmlanguages, numerical analysis, non-English-based languages,object-oriented class-based languages, object-oriented prototype-basedlanguages, off-side rule languages, procedural languages, reflectivelanguages, rule-based languages, scripting languages, stack-basedlanguages, synchronous languages, syntax handling languages, visuallanguages, wirth languages, embeddable languages, and xml-basedlanguages. The memory unit 206 may also be used for storing temporaryvariable or other intermediate information during execution ofinstructions to be executed by the processor unit 202.

A computer program as discussed herein does not necessarily correspondto a file in a file system. A program can be stored in a portion of afile that holds other programs or data (e.g., one or more scripts storedin a markup language document), in a single file dedicated to theprogram in question, or in multiple coordinated files (e.g., files thatstore one or more modules, subprograms, or portions of code). A computerprogram can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiplesites and interconnected by a communication network. The processes andlogic flows described in this specification can be performed by one ormore programmable processors executing one or more computer programs toperform functions by operating on input data and generating output.

According to one aspect of the present disclosure, the wearable displaydevice 100 can be implemented using a computer system 200 in response tothe processor unit 202 executing one or more sequences of one or moreinstructions contained in the memory unit 206. Execution of thesequences of instructions contained in the memory unit 206 causes theprocessor unit 202 to perform the process steps described herein. One ormore processors in a multi-processing arrangement may also be employedto execute the sequences of instructions contained in the memory unit206. In alternative aspects, hard-wired circuitry may be used in placeof or in combination with software instructions to implement variousaspects of the present disclosure. Thus, aspects of the presentdisclosure are not limited to any specific combination of hardwarecircuitry and software.

The term “machine-readable storage medium” or “computer readable medium”as used herein refers to any medium or media that participates inproviding instructions or data to the processor unit 202 for execution.Such a medium may take many forms, including, but not limited to,non-volatile media, volatile media, and transmission media. Non-volatilemedia include, for example, optical disks, magnetic disks, orsolid-state memory such as flash memory. Volatile media include dynamicmemory, such as the memory unit 206. Transmission media include copperwire including the wires that include the bus. The machine-readablestorage medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them.

In one or more implementations, the wearable display device 100communicates via a wireless connection (e.g., electromagnetic radiation,infrared radiation or other wireless technology), a wired connectionusing any type of wire-based communication protocol (e.g., serialtransmissions, parallel transmissions, packet-based datacommunications), or a combination of one or more wireless connectionsand one or more wired connections. Example communications modules (e.g.,the communication unit 204) include one or more radio circuits and/ortransceiver circuits (e.g., transmitter/receiver circuits).

The input device 144 may be used to make selections with respect to agraphical user interface (GUI) on the display units 112 and 122 (FIG.1). The input device 144 may also be used to issue commands to processorunit 202. Multiple kinds of input devices 144 can be used to provide forinteraction with a user, such as a tactile input device, visual inputdevice, audio input device, or brain-computer interface device. Forexample, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including acoustic,speech, tactile, or brain wave input. In some aspects, the input device144 includes a touch sensing element for receiving input from a user'stouch and sending the received input to the processor unit 202. Thetouch sensing element may be a touchpad, which may recognize touchpoints, in terms of coordinates of their positions as well as theirmagnitudes on a corresponding touch sensitive surface. The touch sensingelement may be based on sensing technologies including but not limitedto capacitive sensing, resistive sensing, surface acoustic wave sensing,pressure sensing, optical sensing, and/or the like. Additionally, thetouch sensing element may be configured to detect a single touch pointor a plurality of simultaneous touch points. The input device 144 mayreport data corresponding to the touch points to the processor unit 202,which interprets the reported data such that the processor unit 202 mayinitiate a task in accordance with a particular touch. In other aspects,the input device 144 includes a proximity sensing element for receivinginput when the user's finger is within a predetermined distance from asurface of the proximity sensing element, and sending the received inputto the processor unit 202.

The memory unit 206 includes a vision sensing data application 208, amotion sensing and orientation sensing data application 210, an audiosensing data application 214, an image projection application 216,program data 218, an image editing application 220, and an image andgestures library 221. The memory unit 206 may include, but not limitedto, non-volatile memory, such as read-only memory (ROM, flash memory,etc.), and volatile memory, such as random-access memory (RAM), or thelike.

The vision sensing data application 208 activates and deactivates thefirst camera device 114 and the second camera device 116. The visionsensing data application 208 may convert digital image signals,generated by the first camera device 114 and/or the second camera device116, into image input data which is communicated to the processor unit202.

The motion sensing and orientation sensing application 210 activates anddeactivates the motion sensor devices 134 and 136. The motion sensingand orientation sensing data application 210 may convert analog and/ordigital signals, generated by the motion sensor devices 134 and 136,into motion and orientation input data which is communicated to theprocessor unit 202.

The audio sensing data application 214 activates and deactivates themicrophone device 142, and may convert generated audio signals intoaudio input data which is communicated to the processor unit 202. Theimage projection application 216 activates and deactivates the imageprojection device 132, and may convert image data, from the processorunit 202, into image input data which is communicated to the imageprojection device 132.

In one or more implementations, the applications 208, 210, 214, 216 and220 can operate with the program data 218 on a suitable operatingsystem. Operating systems are generally well known and will not bedescribed in greater detail. By way of example, the suitable operatingsystem may correspond to mobile device operating systems. The suitableoperating system can also be a special purpose operating system, such asmay be used for limited purpose appliance-type computing devices.

As stated above, hand gestures can be used to provide user inputs to thewearable display device 100 to edit an image displayed on a display unit(e.g., the display units 112, 122) of the wearable display device 100.More specifically, a hand gesture can serve as a user interface of thewearable display device 100, providing functionality to trigger theimage editing application 220 and/or to perform particular editingactions in response. In some aspects, other gestures may be interpretedas a user input to the wearable display device 100 to edit an imagedisplayed on a display unit, such as detecting more than one hand wavingin a particular pattern within the FOV of the first camera device 114and/or second camera device 116, detecting the user's hand moving awayfrom the wearable display device 100 relative to the first camera device114 and/or second camera device 116, detecting the user's hand moving inand out of the FOV of the first camera device 114 and/or second cameradevice 116 that collectively form a particular gesture, detecting one ormore discontinuous movements of the user's hand (e.g., stationarypositions (or non-movements) serving as gaps in time between detectedmovements) that collectively form a particular gesture, and the like.

In some aspects, the three specific hand gestures may respectivelycorrespond to 1) enabling the image editing user interface and selectingone of the three types of image adjustment, 2) adjusting the strength ofthe selected adjustment type and 3) ending the image adjustment. In someaspects, a fourth hand gesture may be used for selecting a particulararea of the displayed image to be adjusted. The three specific handgestures may include 1) an elongated (or expanded) index finger of auser's hand in combination with all remaining fingers bent or closed(e.g., a fist with extended index finger to act as a cursor-like pointergesture, 2) a fully closed hand (e.g., all fingers closed), and 3) afully opened hand (e.g., all fingers expanded).

In some implementations, the wearable display device 100 can recognizethat a hand gesture is being or has been made, and may then identify ormatch the recognized hand gesture against an image and gestures library221 (or other list or collection of gesture items) of known handgestures in order to determine one or more actions to take or invoke inresponse to the identified gesture. Thus, the user can interact (e.g.,provide input data and/or commands) with the wearable display device 100by performing or making a hand gesture in a region of space viewable bythe first camera device 114. As such, the viewable region of space forhand gestures of a user can be a forward-looking FOV of the wearabledisplay device 100, within about an arm's length from the front(forward-facing side) of the wearable display device 100. Thecharacterization of “an arm's length” in front of the wearable displaydevice 100 provides a useful example of a hand gesture space or region,but is not intended to limit the possible region of space around thewearable display device 100, in which hand gestures for interaction withthe wearable display device 100 can be used.

In other aspects, the first camera device 114 monitors the FOV for oneor more triggering hand gestures or criteria. For example, the firstcamera device 114 may monitor the FOV for a particular marker, such as aring bearing a known pattern, which may then be used to signal thepresence of a user's hand within a spatial region designated for handgestures. As another example, the first camera device 114 may monitorthe FOV for the appearance of a moveable object or item having a coloror tone that matches or resembles one or more catalogued colors and/ortones considered to be representative of a variety of skin colors.Again, this may indicate the presence of a user's hand within a spatialregion designated for hand gestures. It will be appreciated that othercriteria or events may signal the presence of a user's hand, and triggerhand gesture analysis.

Upon determining that a triggering event has occurred, image data and/orvideo data from the first camera device 114 may be analyzed for thepresence and/or occurrence of a hand gesture. Considering a static handgesture for purposes of illustration, the wearable display device 100may capture image data and/or video data from the field of view of thefirst camera device 114, and then identify a triggering gesture in thecaptured data. The processor unit 202 may then apply an analytical shapetemplate to the triggering gesture or to a region containing thetriggering gesture. More particularly, the analytical shape template maycorrespond to an adjustable graphical outline that can bealgorithmically configured to fit around an identifiable form of thetriggering gesture. Once a sufficient fit to the form of the gesture isobtained, the adjusted analytical shape template may then be used as aplaceholder or proxy for the triggering gesture in a search for a matchor near match against known/stored shapes in the image and gestureslibrary 221 (or other list or collection) of known hand gestures.Accordingly, interaction with the wearable display device 100 via handgestures can be facilitated by creating and maintaining a library ofhand gestures, referred to as the image and gestures library 221, in thememory unit 206.

In some aspects, a search for a matching (or nearly matching) shape maybe carried out by comparing an analytical expression for the adjustedshape template with each of an analytically-expressed shape in the imageand gestures library 221, in which each known hand gesture is associatedwith a respective, analytically-expressed shape. Each comparison maytake the form of a least-squares fitting procedure, for example, inorder to identify a closest match. Other fitting procedures may be usedas well. The search may test for a best match against known shapes inthe image and gestures library 221. Alternatively, the search mayproceed sequentially and terminate upon a first occurrence of a matchthat exceeds a threshold statistical significance, for example. Once amatch (or a best-fit match) is found, the hand gesture can then beconsidered as recognized (or detected) by the wearable display device100, and an associated action or actions are carried out.

By way of example, the image and gestures library 221 may be stored inthe memory unit 206, and accessed by the processor unit 202 as part ofthe search for a matching shape. Each known shape, such as apredetermined hand gesture, in the image and gestures library 221 may bestored with a corresponding gesture identifier (e.g., a name ordescription), as well as indicia of one or more actions, programs, orapplications that may be carried out or invoked upon matching the knownshape against a received, adjusted, analyzed shape template of anobserved (or detected) hand gesture. In operation, the adjustment of theshape template to an observed hand gesture, as well as the fittingprocedure and the search for a match, may be implemented as one or moreprograms or applications, and executed by the processor unit 202.

The image and gestures library 221 may include a collection of suchrecords, each containing a different hand gesture and associatedanalytically-expressed shape. The image and gestures library 221 may becomposed of a collection of such data structure records. Upondetermining an adjusted, analytical shape template of an observed (ordetected) hand gesture, the processor unit 202 may perform a tablelook-up or other form of search in the image and gestures library 221 toidentify a closest match with one of the stored analytically-expressedshapes. In this respect, the observed hand gesture may be identifiedwith a known, stored gesture identifier. The identified hand gesturemay, in turn, be used to signal or trigger an associated action orapplication on the wearable display device 100, for example.

In one or more implementations, the image and gestures library 221 maybe created by generating each hand gesture through “learning” orrecording processes. More particularly, the wearable display device 100may be placed into a recording mode during which a given,initially-undefined (or unassigned) hand gesture is made within aviewable region of space used for hand gestures. The given hand gesturemay then be recorded in the form of an analytically-expressed shape. Insome aspects an associated gesture identifier as well as associatedactions may be assigned to the analytically-expressed shape. Thewearable display device 100 may then exit the recording mode. Thenewly-recorded hand gesture may be stored as an entry in the gesturelibrary. This process may be repeated to increase the inventory of handgestures in the image and gestures library 221. Thereafter, the recordedhand gestures may be available for interaction as described above.Additionally or alternatively, all or part of the image and gestureslibrary 221 may be created externally to the wearable display device100, and pre-loaded during system configuration or manufacture of thewearable display device 100, for example.

In some aspects, the image and gestures library 221 may include at leastone alerting hand gesture, the purpose of which can be to signal (ornotify) the wearable display device 100 to monitor for, or be “on thelook-out” for, one or more subsequent, other hand gestures. The imageand gestures library 221 may include more than one alerting handgesture, each having a different, specific purpose or function, such asadjusting contrast, brightness, or saturation, for example.

The alerting hand gesture may be used, for example, to place thewearable display device 100 in a mode or state in which the wearabledisplay device 100 may expect one or more subsequent hand gestures. As afurther example, an alerting hand gesture may be used by the user of thewearable display device 100 to indicate that the wearable display device100 is observing the user's hand or hands, thereby helping avoidpossible confusion with images of someone else's hands that may becaptured by the wearable display device 100. Additionally oralternatively, an alerting hand gesture may support an authorizationand/or authentication function, in which the image editing operation mayutilize detection of the alerting hand gesture before the wearabledisplay device 100 captures images or video of any subsequent handgestures. For example, a user of the wearable display device 100 maylock and unlock access to a hand gesture function of the wearabledisplay device 100 with an alerting hand gesture that included anauthorization and/or authentication function.

In situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about a user'ssocial network, social actions or activities, profession, a user'spreferences, or a user's current location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. In addition, certain data may be treated in one or moreways before it is stored or used, so that personally identifiableinformation is removed. For example, a user's identity may be treated sothat no personally identifiable information can be determined for theuser, or a user's geographic location may be generalized where locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular location of a user cannot be determined. Thus, theuser may have control over how information is collected about the userand used by a content server.

FIG. 3 illustrates a flowchart of an example process 300 for imageediting using hand gestures detected by the wearable display device ofFIG. 1 according to certain aspects of the disclosure. The process 300may be performed using, for example, the wearable display device 100 ofFIG. 1, particularly, the processor unit 202 of FIG. 2. While FIG. 3 isdescribed with reference to FIG. 2, it should be noted that the processblocks of FIG. 3 may be performed by other systems. FIG. 3 illustratesthe blocks in the example process 300 as shown; however, it isunderstood that in other implementations of the example process 300, theblocks may appear in different order and blocks may be added orsubtracted.

Once a user triggers (or initiates) a display of an image on a displayunit (e.g., the display units 112 and 122) of the wearable displaydevice 100 (302), the processor unit 202 activates the first cameradevice 114 (e.g., the FOV camera) and the vision sensing dataapplication 208 to capture image data and/or video data, which mayinclude a representation of a hand gesture by the user (304). Uponauthorization by the user, the processing of any captured image data maybe initiated and performed by the processor unit 302.

Upon capture of image data by the first camera device 114 within anarm's length from the wearable display device 100, the vision sensingdata application 208 determines whether the captured image data includesan identifiable hand gesture that triggers an editing process of thedisplayed image (306). In this embodiment, the vision sensing dataapplication 208 compares a stored triggering hand gesture, whichincludes an extended index finger for example, to the identifiable handgesture in the captured image data.

Once the processor unit 202 determines that the captured hand gesturematches or nearly matches the stored triggering hand gesture, theprocessor unit 202 triggers a launching of the image editing application220 (308). If the processor unit 202 determines that the captured handgesture does not match or nearly match the stored hand gesture, then theprocess returns to the step of capturing image data and/or video data ofa hand gesture by the user at block 304.

Subsequently, the processor unit 202 provides for display on the displayunit a menu list that includes a number of types of image adjustments(310). The processor unit 202 also may instruct the user to move his orher extended index finger at a position with respect to the first cameradevice 114 that matches a position of a desirable type of imageadjustment (312). For example, the user may hover his or her indexfinger within the FOV of the first camera device 114 such that theposition of the index finger corresponds to the position of a level ofadjustment within the menu, thereby selecting that level of adjustment.In some aspects, the image data includes a sequence of hand gesturemovements such that the amount of change from the start of the sequenceto the end of the sequence translates into a sequence of imageadjustments at a rate of change corresponding substantially to a rate ofchange of the hand gesture movements. For example, if the sequence ofhand gesture movements includes a hand gesture moving from left to rightat a relatively high rate, then the sequence of image adjustments maytransition between the levels of adjustment at a relatively high rate aswell.

Upon detection of a position of the index finger, via the first cameradevice 114, that matches that of the desirable type of adjustment, theprocessor unit 202 instructs the user to move his or her hand or indexfinger to the left or to the right so as to reduce or increase,respectively, a strength of the selected type of adjustment (314). Upondetermination of a movement of the hand or index finger to the left(316), via the first camera device 114, the image editing application220 reduces the strength of the selected type of image adjustment (318).Alternatively, the image editing application 220 increases the strengthof the selected type of image adjustment (320), when the hand or indexfinger is determined to move to the right (316).

In one or more implementations, the image editing application 220enables the user to select a portion of the image to be modified. Forexample, the image editing application 220 may prompt the user to selecta coordinate and/or a quadrant, for example, on the image to be modifiedby the selected type of adjustment as well as by the level of adjustmentselected. In some aspects, the image editing application 220 may promptthe user in addition to the prompt associated with the list ofadjustment types such that separate image information including arepresentation of the user's selection of the image portion would beprovided for processing by the processor unit 202.

In some aspects, the image editing application 220 provides an action,via the processor unit 202, to have the modified image displayed on adisplay unit to allow the user to confirm the adjustment made to theimage. The image may be displayed on the display unit 112 when the imageis modified by the one or more levels of image adjustment type inreal-time. In one or more implementations, the image editing application220 causes the image to be modified by an external processor unit (e.g.,a datacenter or a cloud-based repository). In this respect, the imageediting application 220 sends with the image the selected type of imageadjustment including the selected level of adjustment for the type ofimage adjustment. In turn, the image editing application 220 receivesthe image modified with the requested image adjustments by the externalprocessor unit. In a case where the image is modified by an externalprocessor unit, the user may confirm the adjustment made to the image orrequest one or more additional changes to be made by sending a furtherrequest to the external processor unit with the requested additionalchanges. The image may be stored in the memory unit prior to beingtransmitted to the external processor unit in some embodiments, or sentdirectly to the external processor unit after being captured by thesensor device in other embodiments.

In some aspects, the processor unit 202 provides for display themodified image on a display unit (e.g., the display units 112, 122)during the image adjustments in real-time, or after the image has beenmodified (322). Upon completion of the image editing operation, theprocessor unit 202 stores the modified image in the image and gestureslibrary 221. Alternatively, the processor unit 202 provides fortransmission the modified image to an external repository (e.g., acloud-based database) for storage of the modified image. In someaspects, the processor unit 202 causes the activity of the image editingapplication 220 to terminate after storage of the modified image.Alternatively, the processor unit 202 may further provide for display aninquiry as to whether the user requests to pursue another imageadjustment of the same image or of another image.

As used herein, the term “position” may refer to motion information andposition information respectively including absolute locationcoordinates and relative location coordinates. The location coordinatesmay correspond to a location in a three-dimensional space (e.g., x-axis,y-axis, z-axis). The motion information may describe linear and/orrotational movement within the three-dimensional space such that a rangeof absolute location coordinates (including start and end coordinates)of a moveable object are described. The motion information may describethe moveable object relative to a different device or object other thanthe wearable display device. The position information may describe astationary location of the moveable object within the three-dimensionalspace, including coordinate information of the moveable object relativeto a different device or object other than the wearable display device.

FIG. 4 illustrates an example process 400 for image editing using headmovement detected by the wearable display device of FIG. 1 according tocertain aspects of the disclosure. The process 400 may be performedusing, for example, the wearable display device 100 of FIG. 1,particularly, the processor unit 202 of FIG. 2. While FIG. 4 isdescribed with reference to FIG. 2, it should be noted that the processblocks of FIG. 4 may be performed by other systems. FIG. 4 illustratesthe blocks in the example process 400 initiated at Block 402, however,it is understood that in other implementations of the example process400, the blocks may appear in different order and blocks may be added orsubtracted.

Once a user triggers a display of an image on the display unit of thewearable display device 100, the processor unit 202 activates agyroscope (e.g., the motion sensor device 136) and an accelerometer(e.g., the motion sensor device 134). Upon authorization by the user,the processing of any captured image data may be initiated and performedby the processor unit 302.

Upon detection of head movement data by the gyroscope and/or theaccelerometer (404), motion and orientation application 210 determineswhether the sensed head movement data includes an identifiable headgesture that triggers an editing process of the displayed image bycomparing a stored triggering head gesture to the identifiable headgesture found in the sensed data (406). Once the processor unit 202determines that the captured head gesture matches or nearly matches thestored triggering head gesture, the processor unit 202 triggers alaunching of the image editing application 220 (408).

Subsequently, the processor unit 202 provides for display on the displayunit a menu list that includes a number of types of image adjustments(410). The processor unit 202 may also instruct the user to raise orlower his or her head at an angle that correspond to the listingposition of a desirable type of adjustment with respect to a horizontalreference, for example, determined by the gyroscope. Upon detection ofan angular position of the user's head, via the gyroscope, that matchesthat of the desirable type of adjustment (412), the processor unit 202instructs the user to move his or her head to the left or to the rightso as to reduce or increase, respectively, a strength (or intensity) ofthe selected type of image adjustment (414). In some aspects, the imagedata includes a sequence of head movements such that the amount ofchange from the start of the sequence to the end of the sequencetranslates into a sequence of image adjustments at a rate of changecorresponding substantially to a rate of change of the head movements.For example, if the sequence of head movements indicates that the user'shead is moving at a relatively low rate, then the sequence of imageadjustments may transition between the levels of adjustment at arelatively low rate as well.

Upon determination of a movement of the head to the left, via theaccelerometer (416), the image editing application 220 reduces thestrength of the selected type of adjustment (418). Otherwise, the imageediting application 220 increases the strength of the selected type ofadjustment (420). Upon completion of the image editing operation, theprocessor unit 202 stores the modified image in the image and gestureslibrary 221, and terminates activity of the image editing application220. Alternatively, the processor unit 202 provides for transmission themodified image to an external repository (e.g., a cloud-based database)for storage of the modified image. In some aspects, the processor unit202 provides for display the modified image on a display unit (e.g., thedisplay units 112, 122) during the image adjustments in real-time, orafter the image has been modified (422).

FIG. 5 illustrates an example process 500 for image editing usingaudible commands detected by the wearable display device of FIG. 1according to certain aspects of the disclosure. The process 500 may beperformed using, for example, the wearable display device 100 of FIG. 1,particularly, the processor unit 202 of FIG. 2. While FIG. 5 isdescribed with reference to FIG. 2, it should be noted that the processblocks of FIG. 5 may be performed by other systems. FIG. 5 illustratesthe blocks in the example process 500 initiated at Block 502, however,it is understood that in other implementations of the example process500, the blocks may appear in different order and blocks may be added orsubtracted.

Once a user triggers a display of an image on the display unit of thewearable display device 100, the processor unit 202 activates audiosensing data application 214. Upon authorization by the user, theprocessing of any captured audio data may be initiated and performed bythe processor unit 302.

Upon detection of a voice command or audio command via the microphonedevice 142 (504), the audio sensing data application 214 determineswhether the detected audio data includes an identifiable command thattriggers an editing process of the displayed image by comparing a storedtriggering audio command to the identifiable audio command in thedetected audio data (506). Once the processor unit 202 determines thatthe identifiable audio command matches or nearly matches the storedtriggering audio command, the processor unit 202 triggers a launching ofthe image editing application 220 (508). In some aspects, the storedtriggering audio command is associated with a user profile whichincludes user information such as account information and/or accessprivileges to the wearable display device 100. For example, the userprofile may include a waveform profile that represents the storedtriggering audio command as a multi-dimensional audio signal includingmagnitude, frequency and time as dimensions. In this respect, thewearable display device 100 may support multi-user access by accessinguser profiles locally stored on the memory unit 206 or accessible froman external repository via the communication unit 204.

Subsequently, the processor unit 202 provides for display on the displayunit a menu list that includes a number of types of image adjustments,each of which may be associated with a number, for example (510). Thetypes of image adjustments may be associated with other types ofstatements representing an action to change an adjustment level, such as“up,” “down.” “more,” “less,” “higher,” “lower,” or the like. In someaspects, the statements may include a statement identifying an amount ofchange relative to a current level of an image property (e.g., “increasesaturation by two”). The processor unit 202 may also instruct the userto state the name of the desirable type of adjustment or state theassociated number. Upon detection of a statement by the user, by theprocessor unit 202 via the audio sensing data application 214, thatmatches the name of the desirable type of adjustment or of theassociated number (512), the processor unit 202 instructs the user tostate a numerical value in a range of 1 to 5, in some embodiments, or ina range of 6 to 10, in other embodiments, so as to reduce or increase,respectively, a strength of the selected type of adjustment (514).

In some aspects, the processor unit 202 via the audio sensing dataapplication 214 may compare the identifiable audio command against apredetermined threshold value that represents a magnitude (or volume)and/or a frequency value corresponding to a particular pitch of thestored triggering audio command. For example, if the processor unit 202determines that the volume of the identifiable audio command satisfiesthe predetermined threshold (e.g., being greater than the threshold),then the processor unit 202 may determine a match. Alternatively, theprocessor unit 202 may determine a match if the pitch of theidentifiable audio command is within the acceptable pitch frequencyrange (e.g., being smaller than the predetermined threshold where thepredetermined threshold represents a maximum frequency).

Upon determination of a statement of one of the numerical values in therange of 1 to 5, via the microphone device 142 (516), the image editingapplication 220 reduces the strength of the selected type of adjustmentto the corresponding stated value (518). Otherwise, the image editingapplication 220 increases the strength of the selected type ofadjustment to the corresponding stated value (520). For example, thestrength of the selected type of adjustment may increase to a level ofadjustment in the range of 6 to 10. In some aspects, the pitch of theidentifiable audio command is used to determine the direction of thestrength adjustment in the image adjustment type. For example, if theidentifiable audio command is determined to have a relatively high pitch(e.g., greater than 100 Hertz), then the processor unit 202 increasesthe strength of the selected type of image adjustment. In addition, themagnitude (or volume) detected in the relatively high pitch may furtheradjust the strength in the increasing direction. For example, thestrength of the selected type of image adjustment may be increasedincrementally if the relatively high-pitched audio command becomesprogressively louder. On the other hand, if the identifiable audiocommand is determined to have a relatively low pitch (e.g., less than100 Hertz), then the processor unit 202 decreases the strength of theselected type of image adjustment. In this example, the processor unit202 may decrease the strength decrementally if the relativelylow-pitched audio command becomes progressively louder.

Upon completion of the image editing operation, the processor unit 202stores the modified image in the image and gestures library 221, andterminates activity of the image editing application 220. Alternatively,the processor unit 202 provides for transmission the modified image toan external repository (e.g., a cloud-based database) for storage of themodified image. In some aspects, the processor unit 202 provides fordisplay the modified image on a display unit (e.g., the display units112, 122) during the image adjustments in real-time, or after the imagehas been modified (522).

In accordance with the present disclosure, editing images on thewearable display device 100 can be performed by one or more handgestures, by one or more head gestures, by audible voice commands, andby one or more eye movements. Accordingly, the wearable display device100 is equipped with one or more cameras that can record hand gestures,with one or more gyroscopes/accelerometers that sense head gesturesand/or head movements, with one or more voice capture units that detectvoice commands, and with one or more eye gazing cameras that captureimages of eye movements by the person wearing the wearable displaydevice 100, thereby enhancing the user experience of editing images viathe wearable display device 100.

In an embodiment, a computer-implemented method includes obtaining imageinformation by a sensor unit of a wearable device, the image informationincluding a moveable object associated with a user. The method includesexecuting an application for editing an image by a processing unit ofthe wearable device, the application being executed based on the imageinformation. The method includes providing for display, on a displayunit of the wearable device, a list of image adjustment types associatedwith the application. The method includes selecting at least one imageadjustment type of the list of image adjustment types based on a firstposition of the moveable object. The method includes providing fordisplay, on the display unit, a prompt including levels of adjustmentfor the selected at least one image adjustment type. The method includesselecting a level of adjustment based on a second position of themoveable object. The method includes modifying an image using theselected level of adjustment. The method further includes providing themodified image for storage by a memory unit of the wearable device.

In another embodiment, a non-transitory machine-readable mediumembodying instructions, which when executed by one or more processors,cause a computing device to perform operations. The operations includereceiving, at a processor unit of a wearable device, first audio datafrom a sensor unit of the wearable device, in which the first audio datais associated with a user. The operations include providing for display,on a display unit of the wearable device, a user interface of anapplication for editing an image based on a first user input in thefirst audio data. The operations include providing for display, on thedisplay unit via the user interface, a list of image adjustment types.The operations include selecting at least one image adjustment type ofthe list of adjustment types based on a second user input via the userinterface. The operations include providing for display, on the displayunit via the user interface, a prompt including levels of adjustment forthe selected at least one image adjustment type, in which each of thelevels of adjustment is displayed with a label corresponding to a levelof adjustment. The operations include selecting a level of adjustmentbased on a third user input via the user interface. The operationsinclude modifying an image based on the selected level of adjustment.The operations further include providing the modified image for storageby a memory unit of the wearable device.

In still another embodiment, an apparatus includes a display unit, oneor more sensor units, one or more memory units and one or moreprocessing units configured to execute executable instructions toperform operations. The operations include receiving, at the one or moreprocessor units, first sensor data from the one or more sensor units,the first sensor data comprising first image information relating to amoveable object associated with a user. The operations includeidentifying one or more first movements by the moveable object includedin the first image information, in which the one or more first movementsinclude one or more of a head movement by the user, a hand movement bythe user or a gaze direction by the user. The operations includedetermining that the one or more first movements correspond to a firstgesture associated with a first user input. The operations includeproviding for display, on the display unit, a user interface of anapplication for editing an image based on the first user input. Theoperations include providing for display, on the display unit via theuser interface, a list of image adjustment types. The operations includedetermining a position of the first gesture relative to the list ofimage adjustment types. The operations include determining a matchbetween a position of at least one image adjustment type of the list ofadjustment types and the determined position of the first gesture. Theoperations include selecting the at least one image adjustment typebased on the determined match. The operations include providing fordisplay, on the display unit via the user interface, a prompt includinga plurality of levels of adjustment for the selected at least one imageadjustment type. The operations include receiving, at the one or moreprocessor units, second sensor data from the one or more sensor units inresponse to the prompt provided for display, in which the second sensordata includes second image information relating to the moveable object.The operations include determining one or more second movements by themoveable object included in the second image information, in which theone or more second movements include one or more of the head movement orthe hand movement by the user. The operations include determining thatthe one or more second movements correspond to a second gestureassociated with a second user input. The operations include selecting alevel of adjustment for the selected at least one image adjustment typebased on the second user input. The operations include modifying animage based on the selected level of adjustment. The operations furtherinclude providing the modified image for storage in a data structure ofthe one or more memory units.

As used herein, the phrase “at least one of” preceding a series ofitems, with the terms “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one item; rather, the phrase allows a meaning that includes atleast one of any one of the items, and/or at least one of anycombination of the items, and/or at least one of each of the items. Byway of example, the phrases “at least one of A, B, and C” or “at leastone of A, B, or C” each refer to only A, only B, or only C; anycombination of A, B, and C; and/or at least one of each of A, B, and C.To the extent that the term “include,” “have,” or the like is used inthe description or the claims, such term is intended to be inclusive ina manner similar to the term “comprise” as “comprise” is interpretedwhen employed as a transitional word in a claim.

Phrases such as an aspect, the aspect, another aspect, some aspects, oneor more aspects, an implementation, the implementation, anotherimplementation, some implementations, one or more implementations, anembodiment, the embodiment, another embodiment, some embodiments, one ormore embodiments, a configuration, the configuration, anotherconfiguration, some configurations, one or more configurations, thesubject technology, the disclosure, the present disclosure, othervariations thereof and alike are for convenience and do not imply that adisclosure relating to such phrase(s) is essential to the subjecttechnology or that such disclosure applies to all configurations of thesubject technology. A disclosure relating to such phrase(s) may apply toall configurations, or one or more configurations. A disclosure relatingto such phrase(s) may provide one or more examples. A phrase such as anaspect or some aspects may refer to one or more aspects and vice versa,and this applies similarly to other foregoing phrases.

A reference to an element in the singular is not intended to mean “oneand only one” unless specifically stated, but rather “one or more.” Theterm “some” refers to one or more. All structural and functionalequivalents to the elements of the various configurations describedthroughout this disclosure that are known or later come to be known tothose of ordinary skill in the art are expressly incorporated herein byreference and intended to be encompassed by the subject technology.Moreover, nothing disclosed herein is intended to be dedicated to thepublic regardless of whether such disclosure is explicitly recited inthe above description.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of what may be claimed, but ratheras descriptions of particular implementations of the subject matter.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

The subject matter of this specification has been described in terms ofparticular aspects, but other aspects can be implemented and are withinthe scope of the following claims. For example, while operations aredepicted in the drawings in a particular order, this should not beunderstood as requiring that such operations be performed in theparticular order shown or in sequential order, or that all illustratedoperations be performed, to achieve desirable results. The actionsrecited in the claims can be performed in a different order and stillachieve desirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in the aspectsdescribed above should not be understood as requiring such separation inall aspects, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products. Othervariations are within the scope of the following claims.

The following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separately claimedsubject matter. In one aspect, any of the claims herein may be revisedto depend from any one of the independent claims or any one of thedependent claims. In one aspect, any of the claims (e.g., dependent orindependent claims) may be revised to be combined with any other one ormore claims (e.g., dependent or independent claims). In one aspect, aclaim may be revised to include some or all of the words (e.g., steps,operations, means or components) recited in a claim, a sentence, aphrase or a paragraph. In one aspect, a claim may be revised to includesome or all of the words recited in one or more claims, sentences,phrases or paragraphs. In one aspect, some of the words in each of theclaims, sentences, phrases or paragraphs may be removed. In one aspect,additional words or elements may be added to a claim, a sentence, aphrase or a paragraph. In one aspect, the subject technology may beimplemented without utilizing some of the components, elements,functions or operations described herein. In one aspect, the subjecttechnology may be implemented utilizing additional components, elements,functions or operations.

Methods and operations described in the specification or the claims canbe implemented, for example, using a non-transitory machine-readablemedium, a system or a computer-implemented method. A non-transitorymachine-readable medium may embody instructions, where the instructionsare executable by one or more processors to perform the method(s) oroperation(s). A system may comprise one or more memories and one or moreprocessors configured to execute executable instructions to perform themethod(s) or operation(s). A machine-implemented method may comprise themethod(s) or operation(s).

What is claimed is:
 1. A computer-implemented method, comprising:obtaining image information by a sensor unit of a wearable device, theimage information including a representation of a moveable objectassociated with a user; providing for display, on a display unit of thewearable device, a list of image adjustment types associated with anapplication for editing an image by a processing unit of the wearabledevice; selecting at least one image adjustment type of the list ofimage adjustment types based on a first position of the moveable objectdetected by the sensor unit; providing for display, on the display unit,a prompt including a plurality of levels of adjustment for the selectedat least one image adjustment type; selecting a level of adjustment fromthe plurality of levels of adjustment based on a second position of themoveable object; selecting a portion of the image for applying theselected at least one image adjustment type with the selected level ofadjustment, based on a third position of the moveable object; andproviding an image with the selected portion modified by the selected atleast one image adjustment type with the selected level of adjustment,for display on the display unit.
 2. The computer-implemented method ofclaim 1, further comprising: modifying the selected portion of the imageusing the level of adjustment for the selected at least one imageadjustment type.
 3. The computer-implemented method of claim 1, furthercomprising: receiving, at a processor unit of the wearable device, theimage information as part of first image data from the sensor unit, thefirst image data comprising one or more images of the moveable object;identifying one or more first movements by the moveable object in theone or more images; determining that the one or more first movementscorrespond to a first gesture associated with enabling a user interfaceof the application; and providing for display, on the display unit, theuser interface of the application based on the one or more firstmovements, wherein the list of image adjustment types and the promptincluding the plurality of levels of adjustment, are displayed in theuser interface.
 4. The computer-implemented method of claim 1, wherein:the moveable object relates to a hand of the user, the second positionof the moveable object relates to a hand gesture by the user, and eachimage adjustment type of the list of image adjustment types isselectable by a different hand gesture position.
 5. Thecomputer-implemented method of claim 4, wherein identifying the one ormore first movements by the moveable object includes determining a shapetemplate for each of the one or more first movements, and whereindetermining that the one or more first movements correspond to a firstgesture associated with enabling the user interface includes matchingthe shape templates of each of the one or more first movements withstored shapes of gestures.
 6. The computer-implemented method of claim1, wherein: the moveable object relates to a head of the user, thesecond position of the moveable object relates to a head gesture by theuser, and each image adjustment type of the list of image adjustmenttypes is selectable by a different head gesture position.
 7. Thecomputer-implemented method of claim 1, wherein: the moveable objectrelates to an eye of the user, the second position of the moveableobject relates to a gaze direction of the eye, and each image adjustmenttype of the list of image adjustment types is selectable by a differentgaze direction.
 8. The computer-implemented method of claim 1, whereinselecting the at least one image adjustment type of the list of imageadjustment types based on a first position of the moveable object,further comprises: determining the first position of the moveable objectrelative to the list of image adjustment types; and determining a matchbetween a position of the at least one image adjustment type and thefirst position of the moveable object, wherein the at least one imageadjustment type is selected based on the match determined.
 9. Anapparatus, comprising: a display unit; one or more sensor units; one ormore memory units; and one or more processor units configured to executeexecutable instructions to perform operations to: receive, at the one ormore processor units, first sensor data from the one or more sensorunits, the first sensor data comprising first image information having arepresentation of a moveable object associated with a user; identify oneor more first movements by the moveable object included in the firstimage information, wherein the one or more first movements include oneor more of a head movement by the user, a hand movement by the user oran eye gaze direction by the user; determine that the one or more firstmovements correspond to a first gesture associated with enabling a userinterface; provide for display, on the display unit, the user interfaceof an application for editing an image based on the one or more firstmovements; provide for display, on the display unit via the userinterface, a list of image adjustment types; determine a position of thefirst gesture relative to the list of image adjustment types; determinea match between a position of at least one image adjustment type of thelist of adjustment types and the position of the first gesture; selectthe at least one image adjustment type based on the match determined;provide for display, on the display unit via the user interface, aprompt including a plurality of levels of adjustment for the at leastone image adjustment type selected; receive, at the one or moreprocessor units, second sensor data from the one or more sensor units inresponse to the prompt provided for display, the second sensor datacomprising second image information having the representation of themoveable object; determine one or more second movements by the moveableobject included in the second image information, wherein the one or moresecond movements include one or more second head movements, one or moresecond hand movements, or one or more second eye movements by the user;determine that the one or more second movements correspond to a secondgesture associated with a second user input; select a level ofadjustment for the at least one image adjustment type, based on the oneor more second movements; determine that one or more third movements bythe moveable object included in third sensor data received at the one ormore processor units, wherein the one or more third movements includeone or more of third head movements, third hand movements, or third eyemovements, by the user; select a portion of the image for applying theselected at least one image adjustment type with the selected level ofadjustment, based on one or more third movements; and provide an imagewith the selected portion modified by the selected at least one imageadjustment type with the selected level of adjustment for display on thedisplay unit.
 10. The apparatus of claim 9, wherein the one or moreprocessing units are configured to execute the executable instructionsto perform further operations to modify the selected portion of theimage based on the level of adjustment, wherein the level of adjustmentincludes an action to increase or decrease a strength of the at leastone image adjustment type selected.
 11. The apparatus of claim 9,wherein: the moveable object relates to a hand of the user, the positionof the first gesture relates to a hand gesture by the user, and eachimage adjustment type of the list of image adjustment types isselectable by a different hand gesture position.
 12. The apparatus ofclaim 9, wherein: the moveable object relates to a head of the user, theposition of the first gesture relates to a head gesture by the user, andeach image adjustment type of the list of image adjustment types isselectable by a different head gesture position.
 13. The apparatus ofclaim 9, wherein: the moveable object relates to an eye of the user, theposition of the first gesture relates to a gaze direction of the eye,and each image adjustment type of the list of image adjustment types isselectable by a different gaze direction.
 14. The apparatus of claim 9,wherein identifying the one or more first movements by the moveableobject includes determining a shape template for each of the one or morefirst movements, and wherein determining that the one or more firstmovements correspond to the first gesture associated with enabling theuser interface includes matching the shape templates of each of the oneor more first movements with stored shapes of gestures.